1. Field of the Invention
The present invention relates to a non-intrusive system for testing computer hardware and/or software, and specifically relates to the use of an automated form of testing known as regression testing.
2. Related Art
Computer-related regression testing systems used for testing hardware and/or software applications exist which can receive a series of test steps (input data) such as keystrokes and pointer movements recognized by the software application. Upon receipt of the input data, these systems have the ability to store the input data (into files called test scripts), and to replay them such that the software application functions as though a user were actually entering the input data into the software application. In this way, large amounts of input data can be repeatedly fed into the software application, with a user having only to create a test script by entering the input data only once.
By using such a system, a software application can be executed using a prepared test script, so as to verify that subsequent versions of the software application perform as expected. This can be accomplished by comparing previously stored (or known) results obtained from executing a software application, with results subsequently obtained by replaying the test script through a software application being tested. The fact that the software application being tested performs as expected can also be used as an indication that the hardware on which the software application runs is performing as expected. This ability to repeatedly enter previously stored input data into a software application and compare newly acquired results to some previously stored results is the essence of automated regression testing.
The purpose of regression testing can be better understood through the following explanation. In general, when a software application is normally executed on a computer system, a user enters input data into a computer in order for the user to manipulate the software application in a desired way. For example, in a text editor, the input data typically consists of text, and perhaps certain keystrokes having functions which cause text to be underlined or cause a change in the margins of the text. The object of automated regression testing is to allow a user to enter such input data only once. Thus, this input data is recorded, and can subsequently be played back on the software application over and over again, as if a user were actually entering the input data each time. Certain results or data obtained during the original recording phase are then compared against results or data obtained during a subsequent playback. Again, the purpose to playing back this input data is that different versions of the software application(s) and various hardware components can be tested against the same input data.
From the above, it can be appreciated that regression testing systems have important functions in the software development industry, where during the process of developing a software application, it is often desirable to run each new version of the software application through a long series of test steps. In addition, in the production of computer hardware, it is often desirable to test each manufactured unit by running a software application on the hardware through a series of steps. Manual entry of these steps was traditionally used, but was soon found to be ill-suited for performing regression testing because it was very time intensive, error-prone, and extremely inadequate for production testing or development purposes. In addition, an ability to make performance measurements and compatibility comparisons of the software application in real-time was also desired.
One system for automating the regression testing process was a system jointly developed by Hewlett Packard and Telamon Corporation of Oakland, Calif. This system provides automatic capture of test scripts in the form of keystrokes on a keyboard, movements of a mouse, etc. Automatic playback is provided to replay these test scripts, and this system replays the test scripts as captured on the test application. Automan then compares the data passed from the computer to the terminal, and checks it against the originally input data. However, Automan does not perform comparisons of the screen display. This is not sufficient, since it is still possible for the data to be typed in correctly, but for the screen output to be incorrect.
Some of these earlier systems were fixed pacing systems, which meant that the input data was sent to the software application at a fixed interval, rather than determining when the application was ready to retrieve the next step. Fixed pacing systems such as these were undesirable however, for several reasons. First, it may be important in many instances that each step of the input data reach the software application at the appropriate time (i.e. it is not enough to merely buffer the input data). Second, since feedback from the software application is not being gathered during fixed interval testing, performance measurements of the hardware running the software application cannot be measured.
In addition to the above-mentioned systems, attempts at applying software patches or modifying the software application itself to create performance measurement data were unsuccessful, since this changed the normal behavior of the software application in unpredictable ways. These behavior changes often prohibited accurate performance measurements of the application software.